Industrial manufacturing relies on automatic inspection of objects being manufactured. One form of automatic inspection that has been in common use for decades is based on optoelectronic technologies that use electromagnetic energy, usually infrared or visible light, photoelectric sensors, and some form of electronic decision making.
One well-known form of optoelectronic automatic inspection uses an arrangement of photodetectors. A typical photodetector has a light source and a single photoelectric sensor that responds to the intensity of light that is reflected by a point on the surface of an object, or transmitted along a path that an object may cross. A user-adjustable sensitivity threshold establishes a light intensity above which (or below which) an output signal of the photodetector will be energized.
One photodetector, often called a gate, is used to detect the presence of an object to be inspected. Other photodetectors are arranged relative to the gate to sense the light reflected by appropriate points on the object. By suitable adjustment of the sensitivity thresholds, these other photodetectors can detect whether certain features of the object, such as a label or hole, are present or absent. A decision as to the status of the object (for example, pass or fail) is made using the output signals of these other photodetectors at the time when an object is detected by the gate. This decision is typically made by a programmable logic controller (PLC), or other suitable electronic equipment.
Automatic inspection using photodetectors has various advantages. Photodetectors are inexpensive, simple to set up, and operate at very high speed (outputs respond within a few hundred microseconds of the object being detected, although a PLC will take longer to make a decision).
Automatic inspection using photodetectors has various disadvantages, however, including:
                Simple sensing of light intensity reflected from a point on the object is often insufficient for inspection. Instead it may be necessary to analyze a pattern of brightness reflected from an extended area. For example, to detect an edge it may be necessary to analyze a pattern of brightness to see if it corresponds to a transition from a lighter to a darker region.        It may be hard to arrange the photodetectors when many points on an object need to be inspected. Each such inspection point requires the use of a separate photodetector that needs to be physically mounted in such a way as to not interfere with the placement of the other photodetectors. Interference may be due to space limitations, crosstalk from the light sources, or other factors.        Manufacturing lines are usually capable of producing a mix of products, each with unique inspection requirements. An arrangement of photodetectors is very inflexible, so that a line changeover from one product to another would require the photodetectors to be physically moved and readjusted. The cost of performing a line changeover, and the risk of human error involved, often offset the low cost and simplicity of the photodetectors.        Using an arrangement of photodetectors requires that objects be presented at known, predetermined locations so that the appropriate points on the object are sensed. This requirement may add additional cost and complexity that can offset the low cost and simplicity of the photodetectors.        
Another well-known form of optoelectronic automatic inspection uses a device that can capture a digital image of a two-dimensional field of view (FOV) in which an object to be inspected is located, and then analyze the image and make decisions. Such a device is usually called a machine vision system, or simply a vision system. The image is captured by exposing a two-dimensional array of photosensitive elements for a brief period, called the integration or shutter time, to light that has been focused on the array by a lens. The array is called an imager and the individual elements are called pixels. Each pixel measures the intensity of light falling on it during the shutter time. The measured intensity values are then converted to digital numbers and stored in the memory of the vision system to form the image, which is analyzed by a digital processing element such as a computer, using methods well-known in the art to determine the status of the object being inspected.
In some cases the objects are brought to rest in the field of view, and in other cases the objects are in continuous motion through the field of view. An event external to the vision system, such as a signal from a photodetector, or a message from a PLC, computer, or other piece of automation equipment, is used to inform the vision system that an object is located in the field of view, and therefore an image should be captured and analyzed. Such an event is called a trigger.
Machine vision systems avoid the disadvantages associated with using an arrangement of photodetectors. They can analyze patterns of brightness reflected from extended areas, easily handle many distinct features on the object, accommodate line changeovers through software systems and/or processes, and handle uncertain and variable object locations.
Machine vision systems have disadvantages compared to an arrangement of photodetectors, including:                They are relatively expensive, often costing ten times more than an arrangement of photodetectors.        They can be difficult to set up, often requiring people with specialized engineering training.        They operate much more slowly than an arrangement of photodetectors, typically requiring tens or hundreds of milliseconds to make a decision. Furthermore, the decision time tends to vary significantly and unpredictably from object to object.        
Machine vision systems have limitations that arise because they make decisions based on a single image of each object, located in a single position in the field of view (each object may be located in a different and unpredictable position, but for each object there is only one such position on which a decision is based). This single position provides information from a single viewing perspective, and a single orientation relative to the illumination. The use of only a single perspective often leads to incorrect decisions. It has long been observed, for example, that a change in perspective of as little as a single pixel can in some cases change an incorrect decision to a correct one. By contrast, a human inspecting an object usually moves it around relative to his eyes and the lights to make a more reliable decision.
Also, the limitations of machine vision systems arise in part because they operate too slowly to capture and analyze multiple perspectives of objects in motion, and too slowly to react to events happening in the field of view. Since most vision systems can capture a new image simultaneously with analysis of the current image, the maximum rate at which a vision system can operate is determined by the larger of the capture time and the analysis time. Overall, one of the most significant factors in determining this rate is the number of pixels comprising the imager.
The availability of new low-cost imagers, such as the LM9630 from National Semiconductor of Santa Clara, Calif. that operate at a relatively low-resolution (approximately 100×128 pixels), high frame rate (up to 500 frames per second) and high sensitivity allowing short shutter times with inexpensive illumination (e.g., 300 microseconds with LED illumination), have made possible the implementation of a novel vision detector that employs on-board processors to control machine vision detection and analysis functions. A novel vision detector using such an imager, and overall inspection system employing such a vision detector, is taught in copending and commonly assigned U.S. patent application Ser. No. 10/865,155, entitled METHOD AND APPARATUS FOR VISUAL DETECTION AND INSPECTION OF OBJECTS, by William M. Silver, filed Jun. 9, 2004, and the teachings of which are expressly incorporated herein by reference (herein also termed “above-incorporated-by-reference METHOD AND APPARATUS).
An advantage to the above-incorporated-by-reference detection and inspection METHOD AND APPARATUS is that the vision detector can be implemented within a compact housing that is programmed using a PC or other Human-Machine Interface (HMI) device (via, for example, a Universal Serial Bus (USB)), and is then deployed to a production line location for normal runtime operation. The outputs of the apparatus are (in one implementation) a pair of basic High/Low lines indicating detection of the object and whether that object passes or fails based upon the characteristics being analyzed. These outputs can be used (for example) to reject a failed object using a rejection arm mounted along the line that is signaled by the apparatus' output.
By way of example, FIG. 1 shows an illustrative embodiment of a vision detector 100 according to the above-incorporated-by-reference METHOD AND APPARATUS FOR VISUAL DETECTION AND INSPECTION OF OBJECTS inspecting objects on a production line. A conveyor 102 transports objects to cause relative movement between the objects and the field of view (FOV) of vision detector 100. Objects 110, 112, 114, 116 and 118 are shown. In this example, the objects include exemplary features upon which location and inspection are based, including a label 120 and a hole 124. More particularly, the exemplary vision detector 100 detects the presence of an object by visual appearance and inspects it based on appropriate inspection criteria. If an object is defective (such as the label-less object 116), the vision detector 100 sends a signal via link 150 to a reject actuator 170 to remove the object (116) from the conveyor stream. An encoder 180 operatively related to the motion of the conveyor (or other relative motion) sends a signal 160 to the vision detector 100, which uses it to insure proper delay of signal 150 from the encoder count where the object crosses some fixed, imaginary reference point 190, called the mark point. If an encoder is not used, the delay can be based on time instead.
In an alternate example, the vision detector 100 sends signals to a PLC for various purposes, which may include controlling a reject actuator. In another exemplary implementation, suitable in extremely high-speed applications or where the vision detector cannot reliably detect the presence of an object, a photodetector is used to detect the presence of an object and sends a signal to the vision detector for that purpose. In yet another implementation, there are no discrete objects, but rather material flows past the vision detector continuously—for example a web. In this case the material is inspected continuously, and signals are sent by the vision detector to automation equipment, such as a PLC, as appropriate.
Basic to the function of the vision detector 100 in the above-incorporated-by-reference METHOD AND APPARATUS is the ability to exploit the abilities of the imager's quick-frame-rate and low-resolution image capture to allow a large number of image frames of an object passing down the line to be captured and analyzed in real-time. Using these frames, the apparatus' on-board processor can decide when the object is present and use location information to analyze designated areas of interest on the object that must be present in a desired pattern for the object to “pass” inspection.
With brief reference to FIG. 2, a timeline is shown, which illustrates a typical operating cycle for a vision detector in visual event detection mode. A portion 200 of the exemplary timeline corresponds to the inspection of a first object, and contains the capture and analysis of seven frames by the vision detector. A second portion 210 corresponds to the inspection of a second object, and contains five frames.
Boxes labeled “c”, such as box 220, represent image capture by the vision detector 100. Boxes labeled “a”, such as box 230, represent image analysis. It is desirable that capture “c” of the next image be overlapped with analysis “a” of the current image, so that (for example) analysis step 230 analyzes the image captured in capture step 220. In this timeline, analysis is shown as taking less time than capture, but in general analysis will be shorter or longer than capture depending on the application details. If capture and analysis are overlapped, the rate at which a vision detector can capture and analyze images is determined by the longer of the capture time and the analysis time. This is the “frame rate”. The above-incorporated-by-reference METHOD AND APPARATUS allows objects to be detected reliably without a trigger signal, such as that provided by a photodetector.
Each analysis step “a” first considers the evidence that an object is present. Frames where the evidence is sufficient are called active. Analysis steps for active frames are shown with a thick border, for example analysis step 240. In an exemplary implementation, inspection of an object begins when an active frame is found, and ends when some number of consecutive inactive frames are found. In the example of FIG. 2, inspection of the first object begins with the first active frame corresponding to analysis step 240, and ends with two consecutive inactive frames, corresponding to analysis steps 246 and 248. Note that for the first object, a single inactive frame corresponding to analysis step 242 is not sufficient to terminate the inspection.
At the time that inspection of an object is complete, for example at the end of analysis step 248, decisions are made on the status of the object based on the evidence obtained from the active frames. In an exemplary implementation, if an insufficient number of active frames were found then there is considered to be insufficient evidence that an object was actually present, and so operation continues as if no active frames were found. Otherwise an object is judged to have been detected, and evidence from the active frames is judged in order to determine its status, for example pass or fail. A variety of methods may be used to detect objects and determine status within the scope of this example; some are described below and many others will occur to those skilled in the art. Once an object has been detected and a judgment made, a report may be made to appropriate automation equipment, such as a PLC, using signals well-known in the art. In such a case a report step would appear in the timeline. The example of FIG. 5 corresponds instead to a setup such as shown in FIG. 1, where the vision detector is used to control a downstream reject actuator 170 via signal 150. By considering the position of the object in the active frames as it passes through the field of view, the vision detector 100 estimates the mark time 250 and 252 at which the object crosses the mark point 190 (FIG. 1). Note that in cases where an encoder 180 is used, the mark time is actually an encoder count; the reader will understand that time and count can be used interchangeably. A report 260, consisting of a pulse of appropriate duration to the reject actuator 170, is issued after a precise delay 270 in time or encoder count from the mark time 250.
Note in particular that the report 260 may be delayed well beyond the inspection of subsequent objects such as object 110 (FIG. 1). The exemplary vision detector 100 uses well-known first-in first-out (FIFO) buffer methods to hold the reports until the appropriate time.
Once inspection of an object is complete, the vision detector 100 may enter an idle step 280. Such a step is optional, but may be desirable for several reasons. If the maximum object rate is known, there is no need to be looking for an object until just before a new one is due. An idle step will eliminate the chance of false object detection at times when an object couldn't arrive, and will extend the lifetime of the illumination system because the lights can be kept off during the idle step.
The processor of the exemplary above-incorporated-by-reference METHOD AND APPARATUS is provided with two types of software elements to use in making its decisions: “Locators” that locate the object and “Detectors” that decide whether an object feature is present or absent. The decisions made by both Locators and Detectors are used to judge whether an object is detected and, if so, whether it passes inspection. In one example, Locators can be simply described as a one-dimensional edge detector in a region of interest. The vision detector is configured for locating objects by placing Locators at certain positions in an image where an edge feature of the object can be seen when the object is in the field of view. The Locator can be oriented with respect to the direction the object is moving, and sized to ensure that the edge feature of the object can be located at multiple positions while in the field of view. During analysis, the location of the edge feature of the object within the Locator can be reported, as well as a logical output state that the location is known.
Detectors are vision tools that operate on a region of interest that produce a logical output state that detects the presence or absence of features in an image of the object. The vision detector is configured for detecting features of an object by placing Detectors at certain positions in an image where object features can be seen when the object is located by the Locators. Various types of Detectors can be used, such as Brightness Detectors, Edge Detectors, and Contrast Detectors.
Detectors can be linked to the location of the feature determined by a Locator to further refine the presence detection and inspection of the object. Accordingly, in each frame where the object may be viewed at a different perspective, the location of the object determined by the Locator will be different, and the position of the Detectors in the image can be moved according to the location determined by the Locator. The operation of the vision detector at high frame rates, therefore permits the vision detector to capture and analyze multiple images of the object while it passes through the field of view.
The above-discussion of Locators and Detectors is further illustrated by way of example in FIGS. 3 and 4. FIG. 3, thus, represents an image of the object 110 from FIG. 1, containing label feature 120 and hole feature 124, with superimposed graphics (termed “Photos” in the above-incorporated METHOD AND APPARATUS) representing a region of the frame whose output can be used to base decisions and is displayed (at appropriate time, such as during vision detector setup—see below) as an “image view” on a Human-Machine Interface (HMI) for a user to view and manipulate. See FIG. 1, for example, showing a detachable or remote HMI 194 with Graphical User Interface (GUI) screen 196 and image view window 198 which displays an associated image view (300) of an object (120, for example) within the vision detector's (100) field of view. Accordingly, FIG. 3 represents an image view, showing the object 300 containing an image of a label 310 and a hole 312. The object in this example contains six visible features to be inspected, corresponding to the two exemplary Locators and four Detectors further described below.
The Locator 320 is used to detect and locate the top edge of the object, and the Locator 322 is used to detect and locate the right edge. A Brightness Detector 330 is used to help detect the presence of the object. In this example the background is brighter than the object, and the sensitivity threshold is set to distinguish the two brightness levels, with the logic output inverted to detect the darker object and not the brighter background. Together the Locators 320 and 322, and the Brightness Detector 330, provide the evidence needed to judge that an object has been detected, as further described below. A Contrast Detector 340 is used to detect the presence of the hole 312. When the hole 312 is absent the contrast would be very low, and when present the contrast would be much higher. A Spot Detector could also be used. An Edge Detector 360 is used to detect the presence and position of the label 310. If the label 310 is absent, mis-positioned horizontally, or significantly rotated, the analog output of the Edge Detector would be very low. A Brightness Detector 350 is used to verify that the correct label has been applied. In this example, the correct label is white and incorrect labels are darker colors.
As the object (110 in FIG. 1) moves from left to right through the field of view of the vision detector 100, the Locator 322 tracks the right edge of the object and repositions Brightness Detector 330, Contrast Detector 340, Brightness Detector 350, and Edge Detector 360 to be at the correct position relative to the object. Locator 320 corrects for any variation in the vertical position of the object in the field of view, repositioning the Detectors based on the location of the top edge of the object. In general Locators can be oriented in any direction. A user can manipulate Photos in an image view by using well-known HMI techniques. A Photo can be selected by clicking with a mouse, and its ROI can be moved, resized, and rotated by dragging. Additional manipulations for Locators are described below.
FIG. 4 shows a logic view containing a wiring diagram corresponding to the example setup of FIG. 3. A wiring diagram shows a series of features (termed “Gadgets” in the above-incorporated-by-reference METHOD AND APPARATUS) 420, 422, 430, 440, 450 and 460 being used to inspect objects and interface to automation equipment, and the connections between logic inputs and outputs of the Gadgets. A wiring diagram may be displayed on an HMI for a user to view and manipulate. A display of Gadgets and their logic interconnections on an HMI is called a logic view. A Locator 420 named “Top”, corresponding to Locator 320 in the image view of FIG. 15, is connected to AND Gate 410 by wire 424. Similarly, “Side” Locator 422 corresponding to Locator 322, and “Box” Detector 430, corresponding to Brightness Detector 330, are also wired to AND Gate 410. The logic output of “Box” Detector 430 is inverted, as shown by the small circle 432, and as described above, to detect the darker object against a lighter background. The logic output of AND Gate 410 represents the level of confidence that the top edge of the object has been detected, the right edge of the object has been detected, and the background has not been detected. When confidence is high that all three conditions are true, confidence is high that the object itself has been detected. The logic output of AND Gate 410 is wired to the ObjectDetect Judge 400 to be used as the object detection weight for each frame. Since the logic input to the ObjectDetect Judge in this case depends on the current frame, the vision detector is operating in visual event detection mode. Note, when operating in external trigger mode, an Input Gadget would be wired to ObjectDetect. To operate in continuous analysis mode, nothing would be wired to ObjectDetect.
The choice of Gadgets to wire to ObjectDetect is made by a user based on knowledge of the application. In the example of FIGS. 3 and 4, a user may have determined that detecting just the top and right edges was not sufficient to insure that an object is present. Note that Locator 322 might respond to the label's left edge just as strongly as the object's right edge, and perhaps at this point in the production cycle Locator 320 might occasionally find some other edge in the background. By adding Detector 330, and requiring all three conditions by means of AND Gate 410, object detection is made reliable. In the wiring diagram, Contrast Detector “Hole” 440, corresponding to Contrast Detector 340, Brightness Detector “Label” 450, corresponding to Brightness Detector 350, and Edge Detector “LabelEdge” 460, corresponding to Edge Detector 360, are wired to AND Gate 412. The logic output of AND Gate 412 represents the level of confidence that all three image features have been detected, and is wired to ObjectPass Judge 402 to provide the object pass score for each frame.
The logic output of ObjectDetect Judge 400 is wired to AND Gate 470. The logic output of ObjectPass Judge 402 is inverted (circle 403) and also wired to AND Gate 470. The ObjectDetect Judge is set to “output when done” mode, so a pulse appears on the logic output of ObjectDetect Judge 400 after an object has been detected and inspection is complete. Since the logic output of ObjectPass 402 has been inverted, this pulse will appear on the logic output of AND Gate 470 only if the object has not passed inspection. The logic output of AND Gate 470 is wired to an Output Gadget 480, named “Reject”, which controls an output signal from the vision detector than can be connected directly to a reject actuator 170 (FIG. 1). The “Reject” Output Gadget 480 is configured by a user to perform the appropriate delay (270 in FIG. 2) needed by the downstream reject actuator.
To aid the user's understanding of the operation of the exemplary vision detector 100, Gadgets and/or wires can change their visual appearance to indicate fuzzy logic values. For example, Gadgets and/or wires can be displayed red when the logic value is below 0.5, and green otherwise. In FIG. 4, wires 404 and 472 are drawn with dashed lines to indicate a logic value below 0.5, and other wires, for example wire 424, are drawn solid to indicate logic values equal to or greater than 0.5. One skilled in the art will recognize that a wide variety of objects can be detected and inspected by suitable choice, configuration, and wiring of Gadgets. One skilled in the art will also recognize that the Gadget class hierarchy of the above-incorporated-by-reference METHOD AND APPARATUS is only one of many software techniques that could be used to practice the implementation.
FIG. 5 illustrates a method for implementing Brightness and Contrast Detectors as described in the above-incorporated-by-reference METHOD AND APPARATUS. In one exemplary implementation of a Brightness Detector, the analog output is the average gray level within the region of interest (ROI) for analysis by the vision detector. In an exemplary implementation, a kernel of positive weights 500 is created corresponding to the size and shape of the ROI, and the analog output A is the weighted average gray level
                    A        =                                            ∑              i                                                                    ⁢                                          w                i                            ⁢                              z                i                                                                        ∑              i                                                                    ⁢                          w              i                                                          (        1        )            where wi is the ith weight and zi is the corresponding pixel gray level. In this example, the weights approximate a Gaussian function of distance r from the center of the kernel to the center of each weight,
                              w          ⁡                      (            r            )                          =                  a          ⁢                                          ⁢                      ⅇ                                          -                                  1                  2                                            ⁢                                                (                                      r                    σ                                    )                                2                                                                        (        2        )            so that pixels near the center are weighted somewhat higher than those near the edge. One advantage of a center-weighted Brightness Detector is that if a bright feature happens to lie near the edge of the Detector's ROI, then slight variations in its position will not cause large variations in the analog output. In FIG. 5, a=99, but any suitable value can be used. The value σ is set based on the diameter d of the kernel,
                    σ        =                  b          ⁢                                    (                              d                -                1                            )                        2                                              (        3        )            In example, b=1.0.
In another exemplary implementation, the analog output is defined by the function C(q), which is the gray level such that:
                                                        ∑                              k                ❘                                                      z                    k                                    ≤                                      C                    ⁡                                          (                      q                      )                                                                                                                                        ⁢                          w              k                                                          ∑              i                        ⁢                          w              i                                      =        q                            (        4        )            where q is a percentile chosen by a user. C is the inverse cumulative weighted distribution of gray levels. Various useful values of q are given in the following table:
qC(q)0.0absolute minimum gray level in ROI0.1statistically reliable minimum gray level0.5weighted median gray level0.9statistically reliable maximum gray level1.0absolute maximum gray level
In one example of a Contrast Detector, the analog output is the standard deviation of the gray levels within the ROI. In an exemplary implementation, the array of positive weights 500 is used to compute a weighted standard deviation:
                    A        =                                                                              ∑                  i                                ⁢                                                      w                    i                                    ⁢                                                            ∑                      i                                        ⁢                                                                  w                        i                                            ⁢                                              z                        i                        2                                                                                                        -                                                (                                                            ∑                      i                                        ⁢                                                                  w                        i                                            ⁢                                              z                        i                                                                              )                                2                                                                        ∑              i                                                                    ⁢                          w              i                                                          (        5        )            
In another example, the analog output is given byC(qhi)−C(qlo)  (6)where the q values may be chosen by the user. Useful values are qhi=0.95, qlo=0.05.
FIG. 6 illustrates an exemplary method for implementing an Edge Detector to detect step edges. A step kernel 600 is created corresponding to the size and shape of the ROI, and the intended direction of the edge. For step kernel 600, the ROI is a circle 12 pixels in diameter, and the direction of the edge is 15 degrees from horizontal. The step kernel 600 is an approximation of the first derivative of a Gaussian function of distance t from the edge to the center of each weight,
                              w          ⁡                      (            r            )                          =                              ar            σ                    ⁢                      ⅇ                          -                                                1                  2                                ⁡                                  [                                                                                    (                                                  r                          σ                                                )                                            2                                        -                    1                                    ]                                                                                        (        7        )            
In the implementation of FIG. 6, a=99, but any suitable value can be used. In this example, the equation 7 is used with b=0.5.
The step kernel 600, with values ki, can be considered to be the product of an ideal step edge template ei and a kernel of positive weights wi:
                                          w            i                    =                                                k              i                                                  ⁢                                  ⁢                              e            i                    =                                    k              i                                      w              i                                      ⁢                                  ⁢                              k            i                    =                                    e              i                        ⁢                          w              i                                                          (        8        )            Note that the ideal step edge template values ei are +1 when ki>0, corresponding to the black on white region of step kernel 600, and −1 when ki<0, corresponding to the white on black region of step kernel 600.Define contrast C and weighted normalized correlation R2 of the step kernel and a like-shaped ROI with pixel values zi as follows:
                              v          =                                                    ∑                i                                                                              ⁢                                                w                  i                                ⁢                                                      ∑                    i                                                                                                  ⁢                                                            w                      i                                        ⁢                                          z                      i                      2                                                                                            -                                          (                                                      ∑                    i                                                                                                  ⁢                                                            w                      i                                        ⁢                                          z                      i                                                                      )                            2                                      ⁢                                  ⁢                  C          =                                    v                                                      ∑                i                                                                              ⁢                              w                i                                                    ⁢                                  ⁢                              R            2                    =                                                    (                                                      ∑                    i                                                                                                  ⁢                                                            k                      i                                        ⁢                                          z                      i                                                                      )                            2                        v                                              (        9        )            The contrast C uses the standard formula for weighted standard deviation, and R2 uses the standard formula for weighted normalized correlation, but simplified because for step kernel 600
                                                        ∑              i                                                                    ⁢                                          w                i                            ⁢                              e                i                                              =                                                    ∑                i                                                                              ⁢                              k                i                                      =            0                          ⁢                                  ⁢                                            ∑              i                                                                    ⁢                                          w                i                            ⁢                              e                i                2                                              =                                    ∑              i                                                                    ⁢                          w              i                                                          (        10        )            
An orthogonal step kernel 610 with values ki′ is also created that is identical to the step kernel 600 but rotated 90 degrees. The ratio
                    D        =                                                                      ∑                i                                                                              ⁢                                                          ⁢                                                k                  i                  ′                                ⁢                                  z                  i                                                                                    ∑                i                                                                              ⁢                                                          ⁢                                                k                  i                                ⁢                                  z                  i                                                                                                  (        11        )            is a reasonable estimate of the tangent of the angle between the actual and expected direction of an edge, particularly for small angles where D is also a good estimate of the angle itself. Note that an orthogonal step template 610 doesn't need to be created—the values from the step kernel 600 can be used, but corresponding to the pixels values in the ROI in a different order.
FIG. 7 shows how the values R2, C, and D are used to determine the analog output of an implementation of the Edge Detector. One can be confident that an edge has been detected when three conditions are met:                1. The ROI looks like an ideal step edge, which means that the weighted normalized correlation R2 of the ideal step edge template and the ROI is high;        2. The contrast C is significantly above some noise threshold; and        3. The angle D is small.        
A weighted normalized correlation operation 700 using ROI 710 and step kernel 720 computes R2. A contrast operation 730 using ROI 710 and step kernel 720 computes C, which is converted by fuzzy threshold operation 740 into a fuzzy logic value 742 indicating the confidence that the contrast is above the noise level. Weighted correlation operations 750 and 752, using ROI 710, step kernel 720, and orthogonal step kernel 722, and absolute value of arctangent of ratio operation 760, compute D, which is converted by fuzzy threshold operation 770 into a fuzzy logic value 772 indicating the confidence that the angle between the expected and actual edge directions is small.
A fuzzy AND element 780 operates on R2 and fuzzy logic values 742 and 772 to produce the analog output 790 of the Edge Detector. Note that R2, being in the range 0-1, can be used directly as a fuzzy logic value. The analog output 790 is in the range 0-1, but it can be multiplied by some constant, for example 100, if a different range is desired. Note that the logic output of an Edge Detector is derived from the analog output using the sensitivity threshold that all Photos have.
FIG. 8 illustrates an exemplary method in accordance with the above-incorporated-by-reference METHOD AND APPARATUS for implementing an Edge Detector to detect ridge edges. A ridge kernel is created corresponding to the size and shape of the ROI, and the intended direction θ of the edge. For ridge kernel, the ROI is a circle 12 pixels in diameter, and the direction θ is 15 degrees from horizontal. The ridge kernel 800 is an approximation of the second derivative of a Gaussian function of distance r from the edge to the center of each weight,
                              w          ⁡                      (            r            )                          =                              a            ⁡                          [                              1                -                                                      (                                          r                      σ                                        )                                    2                                            ]                                ⁢                      ⅇ                                          -                                  1                  2                                            ⁢                                                (                                      r                    σ                                    )                                2                                                                        (        12        )            
In FIG. 8, a=99, but any suitable value can be used. In the exemplary implementation, equation 7 is used with b=0.33.
The use of ridge kernel 800 is similar to that for step kernel 600. The contrast C is computed using the same formula, but R2 uses a different formula because the sum of the kernel values is not 0:
                              R          2                =                                            (                                                                    ∑                    i                                                                                                  ⁢                                                                          ⁢                                                            w                      i                                        ⁢                                                                  ∑                                                                                                                                  ⁢                          i                          ⁢                                                                                                                                                                                                                              ⁢                                                                                          ⁢                                                                        k                          i                                                ⁢                                                  z                          i                                                                                                                    -                                                      ∑                    i                                                                                                  ⁢                                                                          ⁢                                                            w                      i                                        ⁢                                          z                      i                                        ⁢                                                                  ∑                        i                                                                                                                      ⁢                                                                                          ⁢                                              k                        i                                                                                                        )                        2                                v            [                                                            (                                                            ∑                      i                                                                                                            ⁢                                                                                  ⁢                                          w                      i                                                        )                                2                            -                                                (                                                            ∑                      i                                                                                                            ⁢                                                                                  ⁢                                          k                      i                                                        )                                2                                      ]                                              (        13        )            Note that this formula reduces to the one used for step edges when the sum of the kernel values is 0.
A different method is used to determine the angle D between the actual and expected edge directions. A positive rotated ridge kernel 810 with values ki+ is created with an edge direction θ+a, and a negative rotated ridge kernel 810 with values ki− is created with an edge direction θ−a. A parabola is fit to the three points
                              (                      0            ,                                          ∑                i                                                                              ⁢                                                          ⁢                                                k                  i                                ⁢                                  z                  i                                                              )                ⁢                  (                      a            ,                                          ∑                i                                                                              ⁢                                                          ⁢                              k                i                +                                              )                ⁢                  (                                    -              a                        ,                                          ∑                i                                                                              ⁢                                                          ⁢                              k                i                -                                              )                                    (        14        )            The x coordinate of the minimum of the parabola is a good estimate of the angle D between the actual and expected edge directions.
FIG. 9 shows how the ridge kernels are used to determine the analog output of an example of an Edge Detector that can detect either step or ridge edges. For ridge edge detection, weighted normalized correlation 900 uses ROI 910 and ridge kernel 920 to compute R2. Contrast 930 uses ROI 910 and ridge kernel 920 to compute C, which is then converted to a fuzzy logic value by fuzzy threshold 940. Correlation elements 950, 952 and 954 use ROI 910 and ridge kernel 920, positive rotated ridge kernel 924, and negative rotated ridge kernel 922 to provide input to parabolic fit 960 to compute angle D, which is then converted to a fuzzy logic value by fuzzy threshold 970.
R2 and the fuzzy logic values are used by fuzzy AND element 980 to produce a ridge analog output 992 for an Edge Detector that can detect ridge edges. For an Edge Detector that can detect either step or ridge edges, the ridge analog output 992 and analog output 990 from a step edge detector 988 can be used by fuzzy OR element 982 to produce a combined analog output 991.
FIG. 10 shows an exemplary implementation of graphical controls that can be displayed on an HMI for a user to view and manipulate in order to set parameters for an Edge Detector in accordance with the above-incorporated-by-reference METHOD AND APPARATUS. A set of graphical controls displayed on HMI 194 (FIG. 1) for setting Gadget parameters is called a parameter view. Name text box 1000 allows a user to view and enter a Gadget's name. Time label 1002 shows the time taken by the most recent run of a Gadget. Logic output label 1004 shows a Gadget's current logic output value, and may change color, shape, or other characteristic to distinguish between true (≧0.5) and false (<0.5). Invert checkbox 1006 allows a Gadget's logic output to be inverted.
Position control 1020 is used to position a Photo in the field of view. Diameter spinner 1022 is used to change the diameter of a Detector. Direction controls 1024 are used to orient an Edge Detector to the expected edge direction. Position, diameter, and orientation can also be set by manipulation of graphics in an image view, for example the image view of FIG. 3.
Edge type checkboxes 1030 are used to select the types of edges to be detected and the edge polarity. Dark-to-light step, light-to-dark step, dark ridge, and light ridge can be selected. Any combination of choices is allowed, except for choosing none.
Jiggle spinner 1040 allows the user to specify a parameter j such that the Edge Detector will be run at a set of positions ±j pixels around the specified position, and the position with the highest analog output will be used. Sensitivity threshold controls 1050 allow the user to set the sensitivity fuzzy threshold of a Photo. Zero-point label 1051 shows value t0 that can be set by zero-point slider 1052. One-point label 1053 shows value t1, which can be set by one-point slider 1054. Analog output label 1055 shows the current analog output of a Photo. The analog output is also shown graphically by the filled-in region to the left of analog output label 1055, which shrinks and grows like a mercury thermometer lying on its side. The filled-in region can be displayed in three distinct colors or patterns corresponding to a first zone 1056 below t0, a second zone 1057 between t0 and t1, and a third zone 1058 above t1.
Contrast threshold controls 1060 allow the user to view the contrast C and set the contrast fuzzy thresholds 740 and 940. These controls operate in the same manner as the sensitivity threshold controls 1050.
Direction error controls 1070 allow the user to view the angle between the actual and expected edge directions D and set the direction fuzzy thresholds 770 and 970. These controls operate in the same manner as the sensitivity threshold controls 1050, except that the thermometer display fills from right-to left instead of left-to-right because lower values of D correspond to higher fuzzy logic values.
FIG. 11 illustrates an exemplary method for implementing a Spot Detector in accordance with the above-incorporated-by-reference METHOD AND APPARATUS. A spot kernel 1100 is created corresponding to the size and shape of the ROI. For spot kernel 1100, the ROI is a circle 15 pixels in diameter. The spot kernel 1100 is an approximation of the second derivative of a Gaussian function of distance r from the center of the kernel to the center of each weight, using equations 12 and 7. In an exemplary implementation, b=0.6.
The use of spot kernel 1100 is similar to that for ridge kernel 800. Weighted normalized correlation R2 and contrast C are computed using the same formulas as was used for the ridge kernel.
FIG. 12 shows how the spot kernel is used to determine the analog output of an example of a Spot Detector. Operation of the Spot Detector is identical to the Edge Detector example shown in FIG. 7, except that angle D is not computed or used. A weighted normalized correlation 1200 uses ROI 1210 and spot kernel 1220 to compute R2. Contrast 1230 uses ROI 1210 and spot kernel 1220 to compute C, which is then converted to a fuzzy logic value by fuzzy threshold 1240. R2 and the fuzzy logic value are used by fuzzy AND element 1280 to produce a spot analog output 1290.
FIG. 13 shows a pair of image views that will be used to describe the operation of Locators according to an exemplary implementation in accordance with the above-incorporated-by-reference METHOD AND APPARATUS. In a first image view 1300 and a second image view 1302 there is one Detector 1310 and one Locator 1312. The reader will understand that the following description of Detector 1310 and Locator 1312 applies generally to any Detector and Locator. The reader will further understand that many alternate methods can be devised for configuring Locators within the scope of this example.
In one example, a Locator searches a one-dimensional range for an edge, using any of a variety of well-known techniques. The search direction is normal to the edge, and a Locator has a width parameter that is used to specify smoothing along the edge, which is used in well-known ways. The analog output of a Locator depends on the particular method used to search for the edge.
In one example, a Locator searches a one-dimensional range for an edge using the well-known method of computing a projection of the ROI parallel to the edge, producing a one-dimensional profile along the search range. The one-dimensional profile is convolved with a one-dimensional edge kernel, and the location of the peak response corresponds to the location of the edge. A interpolation, such as the well-known parabolic interpolation, can be used if desired to improve the edge location accuracy. In another example, an edge can be located by searching for a peak analog output using the edge detector of FIG. 7 or FIG. 9, once again interpolating to improve accuracy if desired.
In another example, a Locator searches a multi-dimensional range, using well-known methods, which may include translation, rotation, and size degrees of freedom. It will be clear to one skilled in the art how to employ multi-dimensional Locators to position Photos in practicing the example, so the following discussion will be limited to one-dimensional Locators, which are preferred due to their simplicity.
Detector 1310 and Locator 1312 can be moved around in the FOV by clicking anywhere on their border and dragging. Detector 1310 has a resize handle 1320 for changing its diameter, and Locator 1312 has a resize handle 1322 for changing its width and range, and a rotate handle 1324 for changing its direction. All Photos can be moved by dragging the border, and have similar handles as appropriate to their operation.
In the example of FIG. 13, a Locator is drawn in an image view as a rectangle with a inside line segment called the plunger 1330. The width of the Locator is along the plunger, and its range is normal to the plunger. A Locator is oriented by a user so that the plunger is approximately parallel to the edge to be found. The rectangle shows the search range, and the plunger shows the location of a detected edge, if any. If no edge is detected, the plunger is drawn in the center of the range.
A Locator has a rail 1332, shown in FIG. 13 as a dashed line, which is coincident with the plunger but extending in both directions to the edge of the image view.
Every Photo can be linked to zero or more locators, up to some maximum number determined by this example. The number of links determines the number of degrees of freedom that the Locators can control. Degrees of freedom include rotation, size, and the two degrees of freedom of translation. In one example, the maximum number of links is two and only the translation degrees of freedom are controlled.
A linkage defines how a Photo moves as the Locator's plunger moves, following an edge in the image. The movements are defined to keep the Photo at a constant relative distance to the rail or rails of the locators to which it is linked. In this example, the linkages are drawn using a mechanical analogy, such that one could actually build a linkage out of structural elements and bearings and the Photos would move in the same way as forces are applied to the plungers.
In FIG. 13 the linkage from Detector 1310 to Locator 1312 includes a rod 1340, which is rigidly attached to Detector 1310 by a post 1342, and to a slider 1344 that is free to move along the rail 1332, but which holds the rod at right angles to the rail. The post is drawn on the border of a Photo such that the rod, if extended, would pass through the center of the Photo and at the closest of the two possible such points to the rail. A Locator's rail is only shown if there are linkages to it.
Every photo has an emitter, a diamond-shaped handle drawn somewhere on the border. For example Detector 1310 has emitter 1350 and Locator 1312 has emitter 1352. A link is created by drag-dropping a Photo's emitter to any point on a Locator. If the link already exists, the drag-drop might delete the link, or another mechanism for deleting might be used. The user may not create more than the maximum number of allowable links from any Photo, nor any circular dependencies. To aid the user during an emitter drag over a Locator, a tool tip can be provided to tell the user whether a link would be created, deleted, or rejected (and why). Dragging a Locator does not change the behavior of its plunger—it stays locked on an edge if it can find one, or reverts to the center if not. Thus dragging a locator while an edge is detected just changes its search range; the plunger does not move relative to the FOV. More generally, dragging a Locator never changes the position of any Photo to which it is linked. Dragging a Locator will adjust the rod lengths as necessary to insure that no other Photo moves relative to the FOV.
Any plunger may be dragged manually within the range of its Locator, whether or not it has found an edge, and any linked Photos will move accordingly. This allows users to see the effect of the linkages. As soon as the mouse button is released, the plunger will snap back to its proper position (moving linked Photos back as appropriate).
In FIG. 13, Detector 1310 is linked to one Locator 1312, and so one translation degree of freedom is controlled. The degree of freedom is normal to the edge direction, which means that it is in the direction of rod 1340. Comparing second image view 1302 with first image view 1300, the plunger 1330 has moved to the right as it follows an edge (not shown) in the image. Note that the position in the FOV of Locator 1312 has not changed, but Detector 1310 has moved to the right to follow the plunger, which is following an edge of an object and therefore following the motion of the object itself. In our mechanical analogy, Detector 1310 moves because it is rigidly attached to rail 1332 by rod 1340, and the rail moves with the plunger.
FIG. 14 shows a pair of image views that will be used to explain the behavior of a Detector linked to two Locators. In a first image view 1440 and a second image view 1402 Detector 1410 is linked to a first Locator 1420 and a second Locator 1430, and so two translation degrees of freedom are controlled. The degrees of freedom are in the direction of first rod 1422 and second rod 1432. Note that the two degrees of freedom are not independent because they are not orthogonal. Handles and emitters are not shown in FIG. 14.
Comparing second image view 1402 with first image view 1400, first plunger 1424 has moved down as it follows a first edge (not shown) in the image, and second plunger 1434 has moved to the left and slightly down as it follows a second edge (not shown). Note that the positions in the FOV of Locators 1420 and have not changed, but Detector 1410 has moved down and to the left to follow the plungers, which is following the edges of an object and therefore following the motion of the object itself. In a mechanical analogy, Detector 1410 moves because it is rigidly attached to first rail 1426 by first rod 1422, and to second rail 1436 by second rod 1432. Note that first slider 1428 has slid to the left along first rail 1426, and second slider 1438 has slid down along second rail 1436. The sliders slide along the rails when two non-orthogonal Locators are linked to a Photo.
If a Photo is linked to two nearly parallel Locators, its motion would be unstable. It is useful to set an angle limit between the Locators, below which the linked Photo will not be moved. This state can be indicated in some way in the image view, such as by displaying the two rods using a special color such as red. The ability to have Locators either at fixed positions or linked to other Locators provides important flexibility. In FIG. 14 neither Locator is linked and so they remain at fixed positions in the FOV, and therefore at fixed positions relative to the illumination, which is often desirable.
FIG. 15 shows a pair of image views 1500 and 1502 that will be used to explain the behavior of a Detector linked to two Locators, where one of the Locators in linked to the other. In a first image view 1500 and a second image view 1502 Detector 1510 is linked to a first Locator 1520 and a second Locator 1530. Second Locator 1530 is also linked to first Locator 1520 via rod 1540, post 1542, and slider 1544. Slider 1544 slides along rail 1522 of first locator 1520. Note that there need be no limit on the number of Photos that can be linked to a Locator; the degree of freedom limit is on the number of links one Photo can have to Locators. In the example of FIG. 15, Detector 1510 is linked to two Locators and is controlled in two translation degrees of freedom. Second Locator 1530 is linked to one Locator and is controlled in one translation degree of freedom. First Locator 1520 is linked to no Locators and remains fixed in the FOV.
The Locators are configured to follow the top and right edges of a circular feature 1550. Comparing second image view 1502 with first image view 1500, the circular feature 1550 has moved down, causing rail 1522 to move down to follow it. This moves both Detector 1510 and second Locator 1530 down. Note that Detector 1510 is at the same position relative to the object, and so is second Locator 1530. This is desirable in this case, because if second Locator 1530 were fixed in the FOV, it might miss the right edge of circular feature 1550 as it moves up and down. Note that this would not be problematic if the edge of an object in the image was a straight line.
First Locator 1520 has no Locator to move it left and right so as to find the top edge of circular feature 1550. The first Locator 1520 cannot link to second Locator 1530 because that would create a circular chain of links, which is not allowed because one Locator has to run first and it cannot be linked to anything. Instead, the motion of the object through the FOV insures that first Locator 1520 will find the top edge. In the example of FIG. 15, the motion is left to right, and due to the high frame rate of a vision detector the object moves only slightly each frame. Eventually, first Locator 1520 will find the top edge, and will do so on a number of frames, depending on the speed of the object, where the top of circular feature 1550 is close to the center of the Locator. On those frames, second Locator 1530 will be positioned properly to find the right edge, and it will move Detector 1510 left and right as needed to keep it in the right position.
FIG. 16 shows an exemplary method, in accordance with the above-incorporated-by-reference METHOD AND APPARATUS, for handling cases where the edge to be found by a Locator does not extend in a straight line, and so placement of the Locator must be fairly precise along the object boundary. This method could be used for first Locator 1520 in FIG. 15 in an application where objects move at very high speed, and so there might be a chance of missing the top edge entirely as the object moves through the FOV. To handle cases like this, Locators have a parameter that can be used to specify the number of parallel sweeps to be made in searching for the edge. The sweeps are spaced apart along the edge by an amount that provides sufficient overlap so that the edge won't fall between the cracks of the sweeps.
Accordingly, FIG. 16 shows a Locator 1600 with four sweeps that has found an edge on the second sweep from the left. Triangular-shaped sweep markers, including example sweep markers 1610 and 1612, are shown outside the dashed-line sweep rectangle 1620 to avoid interference from the locator graphics within. If an edge is not found on any of the sweeps, the Locator reverts to the center of the sweep rectangle (which won't be at a sweep marker for even sweep counts).
FIG. 17 shows how Locators can be used to handle object rotation and size change even in implementations where only two translation degrees of freedom are controlled. The restriction to translation only provides considerable simplicity and transparency for the user, but small object rotation and size changes can still be handled since Photos in different parts of the FOV can translate differently in response to different Locators. Small rotations and size changes are well-approximated by translations within a small region of the FOV, so as long as Photos are linked to at least one nearby Locator, object rotation and size change will look like translation.
Thus, in FIG. 17 a first image view 1700 and a second image view 1702 contain a first Detector 1710, a second Detector 1712, a first Locator 1720, a second Locator 1722, and a third Locator 1724. First Detector 1710 is linked to nearby first Locator 1712 and second Locator 1722, and will be positioned properly even if the object rotates or changes size (as long as the change is not too big). But second Detector 1712 is too far away—a rotation would tend to mis-position second Detector 1712 vertically relative to second Locator 1722, and a size change would tend to mis-position it horizontally relative to first Locator 1720. Third Locator 1724 is used instead of second Locator 1722 to get the vertical position of second Detector 1712, allowing overall object rotation to be handled. The remote first Locator 1720 is used to get horizontal position for second Detector 1712, so the object size should not vary much. If size variation needs to be handled in addition to rotation, one would add a fourth Locator, near second Detector 1712 and oriented horizontally.
Comparing second image view 1702 with first image view 1700, the object (not shown) has moved to the right and rotated counterclockwise, which can be seen by the motion of the Detectors as the Locators follow the object edges. Note that second Locator 1722 and third Locator 1724 are linked to first Locator 1720 so that they stay close to the Detectors.
Having described in detail the setup of Locators and Detectors in accordance with the above-incorporated-by reference METHOD AND APPARATUS, it should be clear that, while effective, the GUI screen of FIG. 10 requires a good degree of user involvement in order to effectively place desired Locators and Detectors. In general, the HMI GUI screen 196 and image view window (198 in FIG. 1), which is used to assist in setup and testing of the vision detector also provides an opportunity many convenient functions of the vision detector 100 to be manipulated by user with relative ease owing to the graphical nature of the GUI. Slide bars to adjust levels, logic wiring diagrams, menus with selector buttons and drag and drop capabilities (among other data-handling and control features) are variously provided by a GUI, and can be combined with real-time images on the screen 196 returned from the vision detector 100.
Thus, in establishing appropriate Locators and Detectors in an image view of an object during setup, the functionality of the GUI can be highly beneficial. It is desirable that the process for setting up such Locators and Detectors be as easy to use and accurate as possible. By arranging functions of the GUI to facilitate automated setup of locators and detectors, the overall performance and ease of use of the vision detector can be greatly enhanced.